Real time environment model generation system

ABSTRACT

A vehicle environment monitoring system is provided that is based on a three-dimensional vector model. The three-dimensional vector model of the vehicle&#39;s environment is generated on the basis of the image data captured by at least one three-dimensional camera. Out of the image data, particular data are extracted for generating the three-dimensional vector model in order to reduce the data volume. For data extraction, a data extraction algorithm is applied that is determined in accordance with at least one parameter that relates to the situation of the vehicle. Therefore, targeted data extraction is performed for generating a three-dimensional model that is particularly adapted for an application that is desired in the current vehicle situation. The applications of the vector model include driver assistance, external monitoring and vehicle control, as well as recording in an event data recorder. In one implementation, a sequence of three-dimensional vector models, representing a three-dimensional space-and-time model, is generated.

RELATED APPLICATIONS

This application claims priority of European Patent Application SerialNumber 08 006 386.0, filed on Mar. 31, 2008, titled METHOD AND DEVICEFOR GENERATING A REAL TIME ENVIRONMENT MODEL FOR VEHICLES, whichapplication is incorporated in its entirety by reference in thisapplication.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to vehicle environment models for driverassistance systems. In particular, the present invention relates togenerating camera-based vehicle environment models for real timeapplications.

2. Related Art

Contemporary vehicles are equipped with a series of sensors. Vehiclesensors include sensors of a first kind for detecting variables that arerelated to the status of the vehicle itself, as well as sensors of asecond kind for detecting variables of the environment of the vehicle.

Examples of first type sensors include sensors for motor rotation speed,brake status, throttle position, vehicle speed and acceleration,steering position, tire pressure, occupant position, seatbelt position,passenger airbag status and others. Sensors of the second type includetemperature sensors, distance sensors, antennae of telematic andnavigation systems and others.

In particular, in modern practice, it is becoming more and more commonto also equip vehicles with cameras. A vehicle can be equipped with asingle or a plurality of cameras mounted at different positions insideand/or outside the vehicle. Cameras mounted inside a vehicle can beemployed for monitoring objects and events occurring inside the vehicle,and are specifically used in vehicles of public transportation, such asbuses and coaches. Cameras for monitoring the environment of the vehicleare specifically designed to capture images of a certain sector of avehicle's environment.

Data obtained from sensors of a vehicle, including cameras, are employedfor a variety of purposes. A basic class of functions, for which it isnecessary to collect and further process sensor data, is the field ofdriver assistance systems. Driver assistance systems known in the artcover a large range of functions. Systems exist that simply provide adriver with particular information, including warning in the case ofpossible emergency situations inside or outside the vehicle. Moresophisticated driver assistance systems enhance a driver's comfort byinterfering with or partly taking over control functions in complicatedor critical driving situations. Examples for the latter class of driverassistance systems are antilock brake systems (ABS), traction controlsystems (PCS), and electronic stability programs (ESP). Further systemsthat are currently under development and do not yet belong to defaultvehicle equipment include adaptive cruise control, intelligent speedadaptation and predictive safety systems.

Furthermore, vehicle sensors are employed for environment monitoring forother purposes than vehicle safety and control, such as for patrolvehicles monitoring parking cars or for recording in event datarecorders. Event data recorders (EDR, also known as “black box”) recordinternal and external information gathered by various sensors of avehicle to enable a reconstruction of events that have occurred in thelast few seconds or minutes immediately before an accident.

FIG. 1 illustrates an example of a vehicle (a car 100) that is equippedwith a plurality of sensors, including a camera 102 a and furthersensors 102 b and 102 c. The car is moreover equipped with an antenna104, which enables reception and transmission of data, including, forinstance, those of a satellite navigation system. Sensor data isforwarded to a processing unit 106, where the data are processed togenerate a response. A response generated by the processing unit 106 mayinclude signals for triggering any driver assistance functions,including those commonly employed by the above mentioned driverassistance systems. In the simplest case, the processing unit issuessignals comprising information to be notified to a driver. Inparticular, notification can be issued for display on a display device108 that is arranged near the driver's seat.

FIG. 2 is a principal block scheme of a conventional driver assistancesystem. Data collected by a plurality of sensors (s₁, s₂, s₃, . . . ,s_(n) in FIG. 2) are fed into processing unit 106. Processing unit 106generates a response on the basis of processing the data received fromthe sensors. The response includes signals that are either forwarded toa device for notifying the driver, or to specific units for particularcontrol functions.

In the particular case of employing cameras, a rather completeinformation of a section of a vehicle's environment can be obtained bycapturing an image. By employing several cameras, a range of a vehicle'senvironment covers all directions, rather than only a particularsection, such as a section in forward direction. Cameras mounted to avehicle include two-dimensional (2D) and three-dimensional (3D) cameras.While 3D-cameras are capable of capturing a three-dimensional image inone shot, by employing two separate optical systems mounted adjacent toeach other, in a similar manner as a stereoscopic view is achieved withthe help of a pair of human eyes, three-dimensional environmentinformation of a moving vehicle can also be obtained by only employing asingle two-dimensional camera. Therefore, additional information fromsensors detecting a moving speed and direction, as well as changes invehicle orientation are employed. On the basis of the additional data,an approximate three-dimensional environment model can be generated, byevaluating changes between two-dimensional images that have beencaptured at subsequent instances of time with respect to said detectedvehicle motion parameters.

Although cameras are generally capable of providing rather completeinformation, the employment of cameras as sensors suffers from having toprovide a large amount of information that is redundant or irrelevant ina particular situation, or in view of the particular purpose for whichthe information is to be used. Accordingly, a large processing time isrequired to process the large volume of data included in images capturedby cameras. Therefore, in the case of employing cameras as sensors of avehicle, the advantage of obtaining more complete information comparedto a case where only specific sensors are employed, goes hand in handwith a drawback of large processing times required for obtaining aresponse to the received data. However, specifically in the case ofdriver assisting systems utilized to issue a warning to avoid athreatening emergency situation, or to trigger active and/or passivecountermeasures, the importance of processing time is crucial.

A possibility of improving the described situation, and achieving lowerprocessing times, may be achieved by performing a pre-processingprocedure of captured image data. In this regard, to reduce the amountof data to be processed for responding, only particular data areextracted from an image. Such pre-processing steps may include, forinstance, filtering, rasterization and vectorisation.

It is, however, a drawback of employing pre-processed information fromimages captured by cameras attached to a vehicle because thepre-processing affects the information included in the images in apredefined, static manner. The pre-processing steps are adapted tospecific requirements in accordance with particular situations in whicha camera is utilized for environment monitoring. Such specificsituations and the respective circumstances under which the capturedinformation is to be utilized, includes, however, a large range havingtheir own very specific and different requirements. For example,environment monitoring by cameras can be used to avoid potentialcollisions when driving on a motorway at a high speed, and in densetraffic. On the other hand, cameras of a vehicle may assist the driverin parking the vehicle, in a small amount of space. While in the firstcase only rather general information concerning other vehicles in theimmediate vicinity of the vehicle are necessary, the time available fortaking a decision is extremely small, i.e., in the range of a fewseconds. In the second case, processing time is not as crucial as in thefirst case, but the assistance will be more helpful, the more completethe available information.

Another case that is particularly influenced by appropriatepre-processing concerns storage in an event data recorder. The storagecapacity of an event data recorder is generally limited. Therefore, thedepth of the history that can be stored in an event data recorder (i.e.,the time period immediately before an accident, for which the event datarecorder can store data) considerably depends on the amount of data tobe stored representing the overall (external and internal data) vehiclesituation at each single instance of time. On the other hand, the eventdata recorder does not require the inclusion of complete imageinformation, but rather only information concerning position, as well asabsolute value and direction of the velocity of the vehicle itself, andall objects and subjects in the immediate neighborhood of the drivenvehicle. Accordingly, a need exists to provide an improved system formore efficiently obtaining environment monitoring information of acamera-equipped vehicle.

SUMMARY

To address the above illustrated problems with existing vehicleenvironment monitoring systems, a method of monitoring the environmentof a camera-equipped vehicle is provided that utilizes athree-dimensional image of the environment of the vehicle. Athree-dimensional image is first captured that represents apredetermined area of the vehicle environment. A data extractionalgorithm is then utilized to reduce the amount of information acquiredin the image capturing step based on at least one parametercharacterizing the vehicle situation. The amount of information acquiredin the image capturing step is then reduced by extracting data from thethree-dimensional image employing the data extraction algorithm. Athree-dimensional vector model of the vehicle environment is thengenerated from the data extracted from the three-dimensional image.

An environment monitoring device is also provided that monitors theenvironment of a vehicle. In one example, the system includes athree-dimensional camera for capturing an image of the environment ofthe vehicle. The image represents a predetermined area of the vehicleenvironment. The environment monitoring system further includes a firstprocessing unit for processing the information required by thethree-dimensional camera. The first processing unit includes a dataextracting unit for extracting data from the image by employing a dataextraction algorithm. The first processing unit further includesdetermining unit for determining the data extraction algorithm to beemployed by the data extracting unit based on at least one parametercharacterizing the vehicle situation. Moreover, the first processingunit includes a model generating unit for generating a three-dimensionalvector model of the vehicle environment from the data extracted from theimage.

According to another implementation of the invention, a vehicleenvironment monitoring system is provided that is based on athree-dimensional vector model. The three-dimensional vector model ofthe vehicle's environment is generated on the basis of the image datacaptured by at least one three-dimensional camera. Out of the imagedata, particular data are extracted for generating the three-dimensionalvector model to reduce the data volume. For data extraction, a dataextraction algorithm is applied that is determined in accordance with atleast one parameter that relates to the situation of the vehicle.Accordingly, the system specifically targets data extraction that isperformed for generating a three-dimensional model particularly adaptedfor an application that is desired in the current vehicle situation. Theapplications of the vector model may include driver assistance, externalmonitoring and vehicle control, as well as recording in an event datarecorder, to which a sequence of three-dimensional vector models,representing a three-dimensional space-and-time model, may be generated.

Other devices, apparatus, systems, methods, features and advantages ofthe invention will be or will become apparent to one with skill in theart upon examination of the following figures and detailed description.It is intended that all such additional systems, methods, features andadvantages be included within this description, be within the scope ofthe invention, and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE FIGURES

The invention may be better understood by referring to the followingfigures. The components in the figures are not necessarily to scale,emphasis instead being placed upon illustrating the principles of theinvention. In the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 illustrates an example of a vehicle including a prior artenvironment monitoring device.

FIG. 2 illustrates in a block diagram the scheme of a prior artenvironment monitoring system.

FIG. 3 schematically illustrates a vehicle having one example of anenvironment monitoring system in accordance with the invention.

FIG. 4 is a block diagram illustrating on example of an environmentmonitoring system in accordance with the invention.

FIG. 5 is a diagram of the first processing unit of the vehicleenvironment monitoring system of FIG. 4.

FIG. 6 is a flow chart illustrating an example of sensor fusion employedin the vehicle environment monitoring system.

FIG. 7 is a flow chart illustrating one example of the overallprocessing of a vehicle environment monitoring system.

FIG. 8 is a flow chart illustrating a further example of the processingof a three-dimensional vector model of a vehicle's environment.

FIG. 9 is a flow chart illustrating one example of an application of athree-dimensional vector model for driver assistance.

FIG. 10A schematically illustrates the storage of a conventional imagebased event data recorder.

FIG. 10B schematically illustrates the storage of an event data recorderin accordance with an embodiment of the present invention.

FIG. 11 schematically illustrates the concept of cyclically overwritingdata in an event data recorder in accordance with an embodiment of thepresent invention.

FIG. 12 illustrates the structure of an object model employed inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION

In a vehicle equipped with a series of sensors that includes at leastone three-dimensional camera, a situation-adapted three-dimensionalvector model of the current vehicle environment situation may begenerated in real time. The model includes all relevant static andmoving objects in a prior certified range of the vehicle's environment,i.e., in a space area surrounding the vehicle. The objects may becaptured by the camera and further sensors and then classified with thehelp of a database and special algorithms. The model represents aninterface that decouples the sensors of a vehicle from the intelligencefor generating activities in response to the information contained inthe data collected by the sensors, and represented by the vector model.A sequence of three-dimensional vector models generated at differentinstances of time represents a three-dimensional space-and-time model.

By using a three-dimensional space-and-time model of the vehicleenvironment rather than data obtained from various cameras and sensorsthemselves, maximum information density is achieved. The real world isthus compressed into a model by getting rid of irrelevant, useless orredundant information in a manner that is specifically targeted for aparticular situation where the model is generated, and a particular aim,for which the model is to be utilized. Accordingly, memory space andprocessing time can be minimized, and the possibilities for interpretingthe information may be improved.

FIG. 3 is a general overview of a vehicle 300 including one example ofan environment monitoring system in accordance with the presentinvention. The system includes a plurality of sensors s₁, s₂, s₃, . . ., s_(n). Among the sensors is at least one three-dimensional camera,which in this example, is represented by 302 a.

In one example of an implementation, a plurality of cameras of differenttypes (not shown) may be included in the environment monitoring system,where different types of cameras cover different viewing angles. Camerasto watch the traffic in front of the car may capture images that can beutilized to assist a driver with pre-crash detection, land departurewarning, traffic sign recognition, distance of objects measurement,active cruise control, etc. Furthermore, an internal camera can beemployed to replace sensors that recognize the number of passengers,their state and behavior, the seatbelt state etc. to control, forinstance, the behavior of airbags.

The three-dimensional camera is generally capable of capturing an imagerepresenting information about a certain sector (i.e., corresponding toa camera's view angle) of the vehicle's environment. A conventionalthree-dimensional camera (e.g., stereoscopic camera) includes twooptical systems at slightly different positions having a small distancewith respect to each other. Three-dimensional information is generatedon the basis of a perspective transformation detected by comparing thetwo-dimensional images captured from the two different optical systems.A conventional three-dimensional camera, however, has certain drawbacks.For instance, it is difficult to reconstruct distance information with astereoscopic three-dimensional camera for objects covering a large areawith very similar grey scale values, such as walls or street surfaces.It will, for instance, be difficult to generate distance information toassist the driver in parking a car in front of a wall of a building onthe basis of a stereoscopic three-dimensional camera.

More sophisticated three-dimensional camera systems include so-calledthree-dimensional TOF (time-of-flight) sensors. A TOF basedthree-dimensional camera captures an image of an object in the samemanner as a conventional camera. Additionally, a distance measurement isperformed on the basis of phase difference between light signals sentfrom the camera to the object and reflected therefrom, and a referencesignal. A correlation receiver detects the light signal reflected fromthe object and correlates the received signal with a reference signalthat is transmitted from the camera via a direct path, over apredetermined distance. By evaluating the phase difference, the distancebetween the camera and the object can be safely determined. Byevaluating the distances between the camera and the plurality of objectscaptured in an image, a three-dimensional representation of an image isobtained. A PMD (photonic mixer device) camera may be employed for thesepurposes.

Optionally, the environment monitoring system includes further sensors302 b, 302 c other than cameras. Sensors 302 b, 302 c may, for example,include a motion sensor, an accelerometer, a temperature sensor, a radarsensor, and infrared sensor, without being limited to these examples. Anantenna 304 can represent part of a navigation system for receivingnavigation information, and can, moreover, be employed for transmittingdata from the environment monitoring system to a remote location. Also,in correspondence with an embodiment, re-transmitted data from a remotelocation can be employed for driver assistance and remote controlfunctions. Further, a display 306 arranged near the position of thedriver of the vehicle is utilized to display, inter alia, notificationsto the driver generated by the environment monitoring system.

The environment monitoring system further includes a first processingunit 308, the operation of which first processing unit 308 will beexplained in more detail below. The data collected by the vehiclesensors s₁, s₂, s₃, . . . , s_(n), including at least three-dimensionalcamera 302 a, are all forwarded to the first processing unit 308. Thefirst processing unit 308 processes particular data extracted from thesensor data in order to generate a three-dimensional environment model.

In accordance with one example of an implementation, the environmentmonitoring system further includes a second processing unit 310. In thisimplementation, the three-dimensional vector model (or, a sequence ofthree-dimensional vector models representing a three-dimensionaltime-and-space model) is forwarded to the second processing unit 310.The second processing unit 310 evaluates the three-dimensional vectormodel for generating responses based on particular aspects of theinformation contained in the vector model. Responses generated by thesecond processing unit 310 include, but are not limited to, driverassistance functions, such as collision warning, distance adaptive speedcontrol, lane keeping support, lane change assistance or automaticemergency braking.

In another example of an implementation (not shown in FIG. 3), thethree-dimensional vector model generated in the first processing unit308 may be forwarded to a remote location via antenna 304 for furtherprocessing (rather than utilizing a second on-board processing unit asillustrated in FIG. 3). Transmission of the three-dimensional vectormodel to a remote location can be accomplished via radio, via a cellulartelephone line or other transmission means known to those skilled in theart.

In yet another implementation, the vector models generated in the firstprocessing unit 308 may be forwarded to an event data recorder (notshown in FIG. 3) for crash investigation and/or other purposes.

FIG. 4 is a block diagram illustrating an overall view of an environmentmonitoring system 400 in accordance with one implementation of theinvention. The environment monitoring system is represented by firstprocessing unit 308 for generating a three-dimensional vector model 404representing at least a part of the vehicle's environment. Firstprocessing unit 308 receives data from a plurality of sensors, referredto as s₁, s₂, s₃, . . . , s_(n) in sensor complex 418 of FIG. 4.Optionally, environment monitoring system 400 further includes sensorfusion unit 408. Sensor fusion unit 408 may be included to performsensor fusion processing (as will be explained in further detail inconnection with FIG. 6 below), if sensor complex 418 includes sensors ofdifferent types.

FIG. 5 is a more detailed internal view of one example of a firstprocessing unit 308 can be seen. As shown in FIG. 5, the firstprocessing unit 308 includes determining unit 502, data extracting unit504 and model generation unit 508. Data extracting unit 504 operates toextract particular data out of the full amount of data received from thethree-dimensional camera, and optionally from further cameras and/orother sensors, in which case, sensor fusion data may be extracted. Onlythat part of data that is extracted by data extracting unit 504 isforwarded to model generation unit 508, and forms the basis forgenerating three-dimensional vector model 404 (FIG. 4). Data that arenot selected for extraction by data extracting unit 504 are discardedand not further processed in first processing unit 308.

According to one implementation of the invention, in first processingunit 308, the algorithm for extracting data in data extracting unit 504is not static, but depends on the vehicle situation at the moment ofgenerating the model. To determine a situation dependent data extractingalgorithm 506, first processing unit 308 includes determining unit 502.Determining unit 502 determines a particular data extracting algorithmto be employed by data extracting unit 504 on the basis of determiningthe current vehicle situation. The vehicle situation is characterized byone or a plurality of parameters, including those describing internalconditions and events of the vehicle, the vehicle's motion status andenvironmental conditions and events in an area surrounding the vehicle.Parameters characterizing the vehicle situation can be obtained from thesensor data received by first processing unit 308, includingthree-dimensional images, and/or sensor fusion data. Furthermore, in oneexample, determining unit 502 receives additional information from othersources, such as status detectors 416 and input interface 402, shown inFIG. 4. The status detector 416 may include a plurality of sensorsdetecting information about the status of the vehicle itself, such asvelocity sensors or accelerometers, brake status sensors, as well ascameras and sensors for detecting the occupation of seats of thevehicles with people and their behavior, seatbelt and airbag status.Data received via input interface unit 402 may include user defineddata, such as defining a particular operation mode of vehicleenvironment monitoring system, requiring a particular data extractionalgorithm 506 to be applied by data extracting unit 504, shown in FIG.5.

On the basis of the data extracted by data extracting unit 504, modelgeneration unit 508 generates three-dimensional vector model 404. In athree-dimensional vector model, objects of the real world arerepresented by way of points, lines and areas. Points are represented bytheir co-ordinates with respect to a definite co-ordinate system. Inaccordance with the invention, a coordinate system fixed to the vehiclemay be employed for the vector model. Coordinates of a point withrespect to the point of origin of the coordinate system are called a“vector”. In the vector model, (straight) lines are defined by thevectors defining their end points. Areas are defined in the vector modelby a plurality of lines, forming a polygon.

It is evident from the general definition of the vector model that theamount of data, and therefore the required memory resources, basicallydepend upon the accuracy for representing lines and areas in the vectormodel. Therefore, the general task to be solved in creating vector model404 is to determine an optimal trade-off between data amount and modelaccuracy. As indicated above, the data amount required to represent thevector model is closely related to processing time for generating andfurther processing vector model 404, as well as required storagecapacity, for instance, in an event data recorder, such as event datarecorder 406 shown in FIG. 4.

Additionally, the vector of a vector model can be associated with objectinformation of a standard object represented by the vector. Standardobjects are objects of predefined classes of an object model, whichclasses of object models will be described in more detail below withreference to FIG. 10.

By employing object information including a reference to a standardobject class, extended objects including a plurality of points, linesand areas can be approximately described by way of only a few vectors,defining, for instance, a particular point of the object, as well asobject size and orientation with respect to the particular point.Further information representing, for instance, the shape of the objectspredefined in the object model, is stored in association with thestandard object class. To associate a vector of the vector model withobject information of a particular object class of an object model, thefirst processing unit 308 is associated with database 412. Database 412pre-stores object information, in accordance with an object model of oneimplementation of the invention.

A person skilled in the art is aware that the amount of information thatcan be represented by an object model, including object information suchas color information, or information about texture of an area, islimited only by the available storage and processing capacity.

For further processing, 3-D vector model 404 generated in firstprocessing unit 308 is forwarded to second processing unit 310. Secondprocessing unit 310 includes logic to evaluate the three-dimensionalvector model and to generate a response signal 414. Response signal 414may be employed for driver assistance, taking active and passivecountermeasures in case of a threatening accident and further purposes,to be described in detail below. According to one example approach,where first processing unit 308 and second processing unit 310, as wellas the respective processing steps performed by units 300 and 310 aredecoupled from each other by three-dimensional vector model 404,processing time can be saved for generating a more targeted response414, on the basis of a three-dimensional vector model 404 includingthose data that are particularly relevant for generating the responsedesired in a specific situation.

Alternatively, or in parallel, data of three-dimensional vector model404 can be saved in event data recorder 406. In the illustrated example,the storing processing of the three-dimensional vector model 404 iscontrolled by history recording unit 410.

As a further alternative, three-dimensional vector model 404 can betransmitted via antenna 304 (FIG. 3) to an external (remote) location,for further processing, in place of, or in addition to, the processingshown in FIG. 3.

A person skilled in the art is aware that the further processing of thethree-dimensional vector model 404 generated by first processing unit308 is not limited to the foregoing examples. Moreover, severalprocedures of further processing can be employed in parallel, orsubsequently, or alternatively, by a particular environment monitoringdevice. A person skilled in the art is moreover aware that the describedfunctional units can be represented by specific hardware units, as wellas by circuitry including standard components, such as a CPU, RAM etc.,appropriately programmed by a program instructing the hardwarecomponents to operate in accordance with the invention.

FIG. 6 illustrates an example concept of sensor fusion 408 that may beapplied, as illustrated in FIG. 4, where data from a plurality ofdifferent sensors are taken into account for generatingthree-dimensional vector models 404. In a plurality of parallel,independent steps 602 ₁, 602 ₂, . . . , 602 _(n), a plurality of sensors(corresponding to sensor complex 418 of FIG. 4) detect data, inaccordance with the particular kind of sensors. Specifically, the atleast on three-dimensional camera 302 a (FIG. 3) for capturing an imageof the vehicle environment is understood to be a sensor s₁, s₂, s₃, . .. , s_(n) represented in FIG. 6. Accordingly, image capturing step 702,is considered as a particular example of detection step 602 _(k) (k=1 .. . n). Subsequent step 604 receives the data detected by the pluralityof sensors s₁, s₂, . . . , s_(n), and performs a fusion of the sensors.

Sensor fusion processing combines sensory data from different sources.Data delivered from different sensors, attached to the vehicle aretransformed into normalized environment information data in sensorfusion unit 408. During sensor fusion, information obtained fromdifferent sensors of sensor complex 418 are combined in a manner thatincludes optimization, so that the data set resulting from sensor fusionmay be improved compared to a pure agglomeration of the data of theindividual sensors. Specifically, sensor fusion includes a redundancycheck of the information received from different sources, as well as acheck for inconsistencies. In the case of detected inconsistencies, aprobabilistic approach is generally employed to eliminate theinconsistencies. A person skilled in the art is aware of a plurality ofmethods and algorithms used in sensor fusion, including, but not limitedto Kalman filtering and Bayesian networks. In particular, in the case ofplural three-dimensional cameras, together covering an environment rangearound the vehicle, the result of the sensor fusion contains asynthesized camera scene of the environment surrounding the vehicle 300(FIG. 3).

FIG. 7 is a flow chart illustrating an example of the basic processing700 of a method performed by an environment monitoring system of avehicle in accordance with the invention. In a first step (702), atleast one image is captured by a three-dimensional camera. As indicatedabove, “three-dimensional camera” includes different kinds of devices,such as a conventional three-dimensional camera having two opticalsystems, as well as a camera having a time-of-flight sensor.

Subsequent step 704 determines an extraction algorithm for extractingdata from the captured image to reduce the amount of data to beprocessed for generating a three-dimensional vector model. A single or aplurality of parameters that describe the situation of the vehicle maybe employed for determining an extraction algorithm. Step 704 canextract parameter information from a captured image itself, as well asfrom other sources. Other sources include the result of step 706 ofdetecting status information. Status information detected in step 706and employed by extraction algorithm determining step 704 can include,for instance, a vehicle velocity. Vehicle velocity is an example of aparameter that can be used to determine an extraction algorithm 506(FIG. 5), as vehicle velocity influences the desired accuracy, visualangle and distance of the information to be represented by thethree-dimensional vector model 404 (FIG. 4).

Alternatively, the extraction algorithm determining step 704 can employuser instructions input in step 708 as a parameter for determining aparticular extraction algorithm. User instructions in 708 may include aselection between different operation modes of the environmentmonitoring device, in accordance with particular positions of a switchetc., where a parameter value determining an extraction algorithm isrepresented by a signal indicating the particular switch position of aninput unit 402.

A parameter for determining an extraction algorithm based on capturedimage information itself, is represented by information regarding aspecific object class, such as a person occurring in the vehicle'senvironment. If a person suddenly occurs in the vehicle environment, thedata extraction algorithm has to concentrate on the information of theperson and the surrounding area, in order to determine whether theperson is about to interfere with the driving path of the vehicle, byevaluating subsequently generated three-dimensional vector models.

Further examples for employing image information for deciding a dataextraction algorithm include judging the importance of particularinformation. For instance, in the case of driving in a town, an objectthat is recognized as a traffic light can be marked as to be extractedwith high priority.

It is to be further noted that along with pure image information, oralternatively, extraction algorithm determining step 704 can alsoreceive a sensor fusion result described with reference to FIG. 6, asindicated by item B in FIG. 7.

In accordance with one example of an implementation of the invention,step 704 may determine an extraction algorithm by adjusting a parameterof an algorithm that includes variable parameters. Alternatively, or incombination, step 704 can determine a particular data extractionalgorithm by selecting out of a predetermined plurality of differentdata extraction algorithms.

Subsequent step 710 performs data extraction in compliance with the dataextraction algorithm determined in step 704. Data extraction algorithmscan include, but are not limited to, image filtering on the basis ofadjustable filter parameters, determining a part of the visual angle ofa camera as a basis for extracting data, and extracting data only fromparticular ones of a plurality of cameras/sensors. Moreover, dataextraction can be performed on the basis of object recognition. If anobject is recognized that belongs to a particular object classpre-stored in an object model contained in database 412 (FIG. 4), datato be extracted from an image can be limited to the extraction of anoverall position, size and possibly orientation information from theimage. Up to a certain accuracy, the remaining information can beextracted from the standard object model, but does not need to beprocessed directly from the image. Still alternatively, the extractionof image information can be restricted to a particular range surroundingthe vehicle, in order to achieve enhanced model accuracy within alimited range, while disregarding undesired information outside thelimited range. For instance, in the case when the environment monitoringsystem is operated for a parking assistant, quite accurate informationis desired of a particular portion of the vehicle's environment, namelythe parking site and surrounding objects, such as other vehicles, walls,trees, traffic signs, and possibly people. However, more distant objectscan be disregarded, even if they are captured by the camera. Theexamples given above show that, according to the present invention, abasis of data for generating a vector model is created situationdependent and targeted at the goal to be achieved by further processingthree-dimensional vector model 404 (FIG. 4).

As indicated above, in operation, a PMD-based TOF-camera may be utilizedfor the present invention. Further, a PMD sensor array of m×n (e.g.,16×64) infra-red sensitive pixels may be utilized employed, each ofarray of which registers a distance value corresponding to the incidentlight.

In particular, such a PMD sensor does not only recognize a thusgenerated image of distances, but is moreover capable of recognizingpredetermined objects and return the objects to a processing devicereading out the sensor. Such objects are represented by groups of pixelsof same distance which are calculated in the sensor itself.

A further example employing a multi-stage data reduction is therecognition of traffic signs. Traffic signs are firstly recognized asobjects by the PMD-sensor, since they represent a pixel group thatapproaches the sensor with the (negative) velocity of the vehicle. After(and only after) an object has been recognized to be most likely atraffic sign, a second step of recognition is performed. In this regard,the interesting region, which is determined in relation to object sizeand form, is captured by the camera (CMOS camera) with the bestavailable resolution to determine the exact form and contents of thetraffic sign. This may be achieved by classification algorithms andemploying a database of possible traffic signs. Accordingly, the amountof data may be step-wisely reduced by a method of exclusion: The PMDsensor reduces the whole image by only returning the relevant area for arecognized object, such that only a particular portion of the CMOS imagehas to be investigated in detail. The determined region of interest isquickly reduced to a small group of possible signs by the classificationalgorithm employing the database on the basis of the form (and possiblycolor) to determine the sign.

Three-dimensional vector model 404 is generated in subsequent vectormodel generation step 712. Creating a vector model on the basis of dataextracted from an image is generally based on recognizing discretephysical objects from the image data. Objects of an image are generallydistinguished by a zone, where all pixels of similar color or grey scalevalues. On the other hand, object boundaries correspond to portions inan image where pixel values change abruptly. Before tracing the outlineof objects, spatial filtering algorithms may be applied to the image soas to compensate for noise effects etc. Subsequently, the vectorsrepresenting vertices, lines, and boundary surfaces of an object aredetermined for the vector model.

If the vector model further includes object specification informationassociating an object with a standard object class of an object model,step 712 further includes recognizing that a physical object detectedfrom the image information represents an instance of a particular objectclass. If it is detected that an object represented in the image to bereflected in three-dimensional object model 404 (FIG. 4) belongs to oneof the standard classes of an object model, that are, for instance,pre-stored in database 414 (FIG. 4), a vector representing a particularpoint (for instance, a vertex) of the object is associated in step 712with object specification information of the particular object class. Inthe case of extended objects, coordinate vectors of the particular pointmay be associated with further information, indicating, for instance, anorientation and overall size of the object. The additional informationcan be represented by further vectors, such as a vector indicating theoverall perimeter of the object, or an orientation angle of the objectwith respect to the coordinate axes. Regarding object dimensions, aparticular object class may alternatively include sub-classescorresponding to objects of the same main class, but having differentsizes.

Steps of further processing three-dimensional object model 404 generatedin step 712 are illustrated with reference to FIG. 10. A link betweenFIG. 7 and FIG. 8 is shown by item A in FIGS. 7 and 8, representing acontinuation in the illustrated method or process from FIG. 7 to FIG. 8.

FIG. 8 illustrates a plurality of further processing paths for thethree-dimensional object model 404 generated in accordance with theprocessing method illustrated in FIG. 7. It is noted that processingpaths described below in connection with FIG. 8 are not limited to onlythe illustrated processing paths. The examples provided are offered forillustrative purposes only and are not meant to be limiting. It shouldalso be noted that in different implementations individual process pathsor combinations of processing paths can be applied in parallel orapplied subsequently. It should be further understood that while furtherprocessing of a single vector model is possible, generally a sequence ofthree-dimensional vector models representing a development of thevehicle's environment with time (i.e., three-dimensional space-and-timemodel) may undergo the described further processing.

According to the processing path illustrated on the left hand side ofFIG. 8, three-dimensional vector model 404 is forwarded to secondprocessing unit 310 (FIGS. 3 & 4). In step 802, second processing unit310 performs vehicle internal further processing of the vector model404. According one example, internal further processing ofthree-dimensional vector model 404 performed by second processing unit310 in step 802 includes, but is not limited to an evaluation ofdistances between vehicle 300 (FIG. 3) and other objects. Changes in thedistances between vehicle 300 and other objects with time are evaluatedin a sequence of three-dimensional object models to detect, at an earlystage, a possible emergency situation, such as a threatening collision.If the threatening emergency situation is detected at an early stage,appropriate counter-measures can be taken to either avoid it or tominimize the consequences. Particularly effective response measures canbe taken upon evaluation of three-dimensional vector model 404 if thevector model further includes object specific information on the basisof an object model. Thus, specifically adapted measures can be taken inresponse to changes in a vector model with time concerning, forinstance, a person being about to interfere with the driving path of thevehicle, a sudden distance change between vehicle 300 and other vehiclesbeing driven in the same direction, such as those changing from aparallel lane into the driving lane of the vehicle 300, or to avoid acollision with a static object such as a tree.

In FIG. 8, possible countermeasures taken in response to furtherevaluation of three-dimensional vector model 404, or a sequence thereof,are summarized in step 804. An implementation example of step 804 isdescribed below in more detail, in connection with FIG. 9.

With specific reference to FIG. 9, in step 902, the vehicle status isdetermined on the basis of vehicle status information. Vehicle statusinformation is received from, for example, vehicle status detectors,such as those described above in connection with item 416 of FIG. 4. Insubsequent step 904, the probability of a threatening emergencysituation is evaluated. To evaluate the threat, step 904 may take intoaccount vehicle status information obtained in step 802 (FIG. 8),together with the environment information extracted from thethree-dimensional vector model (e.g., a sequence of three-dimensionalvector models) in further processing step 802. Subsequent judging instep 906 decides whether the probability of an emergency is judged to below, medium or high (i.e., rates the emergency). It has to be understoodthat the general judgment results, “low”, “medium” and “high”, forexample, indicated near three illustrated outgoing paths of box 906 maycorrespond to well-defined values. For instance, “low” probability maycorrespond to a probability less than about 30%, “high” probability to apercentage value of more than about 80%, and “medium” probability to arange from about 30% to about 80%. These percentage values are, however,given by way of an example only, as these values are not essential tothe invention. Other boundaries and ranking systems between theprocessing paths may be utilized. Moreover, the number of processingpaths judged in step 906 is not limited to three. More than threeprocessing paths may be applied, similarly, less than three, for exampletwo, processing paths may be outgoing of step 906.

According to the example illustrated in FIG. 9, the processing flowfurther develops to step 908 in the case of a low evaluation level, tostep 910 in the case of medium evaluation level, and to step 912 in thecase of high probability of an emergency to be expected. In the case oflow probability of emergency, the environment monitoring system issues anotification to the driver of a vehicle, including a warning regardingthe threatening emergency situation. The notification can be issued invarious ways. For example, an acoustic signal may be issued, or anotification is displayed on a display device located within the fieldof view of the driver. Issuing the notification in the form of a displaymay be particularly useful, as details of the threatening emergency canbe provided in an especially easy manner. An acoustic warning may alsobe helpful in safely notifies an inattentive driver. Such an acousticnotification can be made distinctive, by employing several differentsounds, or a combination of acoustic and visual notification is alsopossible.

In the case of medium probability of an emergency, at step 910, anotification is issued that includes some more specific assistance to adriver. The assistance may include, for instance, a proposal for ameasure to be undertaken to avoid a collision. Corresponding measuresinclude, for instance, to reduce speed, or to change lane by slightlyvarying the steering direction. Step 912, applied in the case of a highprobability of emergency in accordance with the example illustrated inFIG. 9, actively interferes with the vehicle control. Therefore, secondprocessing unit 310 generates a signal to be forwarded to a controller(not shown in FIG. 4), which interprets the signal in order to carry outactive measures in compliance with the signal information. For instance,an emergency braking can be triggered to avoid a rear-end collisionaccident.

The range of response actions to information retrieved from evaluationof a three-dimensional vector model (i.e., three-dimensionalspace-and-time model) is not limited to the examples given above. Alarge variety of other responses is possible. For instance, so-calledpassive countermeasures can be carried out, such as airbag control,seatbelt control and pedestrian protection control. For instance,internal safety systems of a vehicle such as seatbelts and airbagsgenerally undergo irreversible processes if fully deployed. Accordingly,these safety systems have to be replaced in the case of full deployment,even if a threatening accident still could have been avoided, andtherefore the vehicle itself has not been damaged. It is thereforeuseful to perform some reversible pre-processing of vehicle safetysystems, such as deployment of reversible seatbelt pretensioners, incase an accident occurs with a certain probability, in order to minimizefull deployment time, if the accident occurs, but not to irreversiblydamage the safety systems in case the accident still could be avoided.For instance, in accordance with the scheme of FIG. 9, reversibleseatbelt pretensioners can be deployed, if step 906 judges a mediumprobability of impact, while the seatbelt is irreversibly locked, ifstep 912 judges a high probability.

An example of a pedestrian protection control system relying oninformation evaluated from the three-dimensional vector model inaccordance with the invention is given below. The example pedestrianprotection control system includes an active system that increases theattack angle of the engine hood of the vehicle to lessen theconsequences for a pedestrian in the case of a collision with the frontof the vehicle. A prerequisite is that a situation, where a pedestrianthreatens to collide with the front of vehicle, is clearly detected andevaluated by the environment monitoring system.

Returning to FIG. 8, in accordance with the second path in the centralposition of the figure, in step 806 a vector model 404 (FIG. 4)generated in first processing unit 308 (FIGS. 3 & 4) is saved into eventdata recorder 406 (also known as a “black box”). A sequence of vectormodels may then be saved. The saving processing is controlled by therecording unit 410.

An event data recorder 406 saving a sequence of generatedthree-dimensional vector models 404 may be utilized as an alternative toa conventional black box that saves sensor data, In particular, whenutilized on a camera equipped vehicle, raw video data may be stored bythe event data recorder 406. On the one hand, the sequence ofthree-dimensional vector models 404 that is generated independently ofthe event data recorder 406, for other purposes, such as driverassistance and/or safety countermeasures, includes all importantparameters necessary for reconstruction of an accident. These parametersinclude, but are not limited to size, kind, speed, direction anddistance data of all objects over space and time relative to thevehicle. On the other hand, in view of the fact that a three-dimensionalvector model 404 includes only extracted data that has been extracted bya data extraction algorithm, a considerably increased spatial andsemantic density may be achieved compared to a conventional event datarecorder, saving complete raw video scenes. Accordingly, while, forevery instant of time, much less storage capacity than in a conventionalevent data recorder is required to store the relevant data for theconstruction of an accident, for a given overall storage capacity, thehistory time period available for accident reconstruction (“historydepth”) can be considerably increased. A comparison of the history depthbetween a conventional camera-based event data recorder and atraditional event data recorder in will be described further below inconnection with FIGS. 10A and 10B.

FIG. 10A schematically illustrates a storage medium of a conventionalevent data recorder. The overall width of the scheme, indicated bydouble-sided arrow 1006 corresponds to the fixed overall storagecapacity of the event data recorder 406. A sequence of images 1002,captured by at least one camera at predetermined instants of time 1004is subsequently stored in respective portions of the storage medium withtime stamps indicating the predefined instants of time 1004. Forsimplicity of illustration, only a single image is indicated to bestored for each instant of time. However, the prior art event datarecorder, as illustrated, is not limited to the case of a single imageto be stored, but a plurality of images captured by different two-and/or three-dimensional cameras can be stored, possibly together withdata received from other sensors, at every instant of time as well. Timestamps t₀, t₁, t₂, t₃, . . . , t_(m) indicate time instances 1004 inincreasing manner, i.e., so that t₀<t₁<t₂<t₃< . . . <t_(m). Asillustrated, time instances for saving the images may be chosen to beequidistant. The overall number of subsequent images that can be storedin an event data recorder as illustrated in FIG. 10A is given by overallstorage capacity 1006. Namely, if the first image has been stored attime instance to, and the overall storage capacity 1006 has been reachedafter storing the image at time instance t_(m) as indicated in FIG. 10A,a total of m+1 images can be stored. Referring to the time scale, thedifference between the first and the last image to be stored, i.e., thehistory depth as defined above, corresponds to the time differencebetween the last and the first time instance, i.e., t_(m)−t₀.

FIG. 10B schematically illustrates a storage medium of an event datarecorder, where a sequence of vector models 404 in accordance with theinvention are stored. The overall storage capacity 1006 of the storagemedium is assumed to be the same as in a conventional storage medium ofFIG. 10A. In contrast to the example of FIG. 10A, in the event datarecorder of FIG. 10B, for every time instance 1004, a correspondingvector model 404 is stored, rather than a single or a plurality of rawimage data 1002 corresponding to the respective time instances 1004. Asindicated above, the required storage volume per time instance isconsiderably lower in the case of vector model 404, and in the case ofraw image data 1002. Accordingly, data of larger number time instancescan be stored in the storage medium, as compared with the conventionalcase. In the example of FIG. 10B, for simplicity, it is assumed that thesame time intervals between time instances for storing (e.g.,equidistant) are employed than in FIG. 10A above.

Consequently, the number of vector models 404 that can be stored for thesame overall storage capacity 1006 is larger than the number of images1002 that can be stored in the conventional event data recorder of FIG.10A. The respective relationship is illustrated by inequality n>m inFIG. 10B, where t_(n) being the instance of time for the last vectormodel that can be stored within storage capacity 1006, after storage hasbeen started at time instance t₀. In the illustrated case of equidistanttime instance, consequently, history depth t_(n)−t₀ of FIG. 10B islarger than history depth t_(n)−t₀ of FIG. 10A.

An possibility to further reduce the amount of data consists inadditionally applying software compression algorithms such as MPEG. Inparticular, objects which remain included in the 3D environment overtime can be reduced along the time line in that not each image needs toinclude the whole information about the object. In each subsequentimage, a reference to an initial image including the whole informationabout the object would be sufficient.

FIG. 11 illustrates a concept of cyclically overwriting entries in thestorage of an event data recorder storing vector models 404. Event datastorage 1100 corresponds to the data storage as illustrated in FIG. 10B.However, for simplicity, the double arrow 1006 indicating the overallstorage capacity has been omitted in FIG. 11. The scheme on top of FIG.11 corresponds to the situation as illustrated in FIG. 10B, where n+1vector models 404 correspond to time instances t₀ to t_(n), have beenstored in the storage. Accordingly, no free space is available forstoring the next vector model 404 to be stored, at time instancet_(n+1). In other words, the maximum available recording time (i.e.,history depth) has been reached. As indicated by the arrow between thetop and bottom scheme of FIG. 11, at a time point t+_(n+1) vector model404 having the earliest time stamp, t₀, is deleted and the respectivestorage portion is occupied by vector model 404 with time stamp t_(n+1).In other words, at time t_(n+1) the vector model having time stamp to isoverwritten by the vector model with time stamp t_(n+1). Accordingly,the overall number of stored vector models, and at the same time, underthe assumption of equidistant time stamps, the history depth remainsconstant: t_(n+1)−t₁=t_(n)−t₀=n(t₁−t₀).

In one example of an implementation, the concept of cyclicallyoverwriting the vector models stored in the data recorder 406 may beapplied as long as no emergency situation is registered. In the case ofan emergency situation, such as an accident leading to an impact beingdetected, history recording unit 410 receives the signal to immediatelystop further recording. Consequently, it is guaranteed that in the caseof an accident really occurring, history information in the form ofvector model 404 is always available for a time period corresponding tohistory depth t_(n)-t₀, immediately before the occurrence time of theaccident.

A problem may occur, if the sensor does not properly recognize an objectsuch that the 3D vector model is incomplete or wrong. The problem may bereduced by including an additional, very small ring buffer that storesfew images directly from the video cameras (for instance only every 2 to5 seconds). These images could be employed for verifying the vectormodel. Since the additional buffer is very small, the data reduction isstill considerable as compared to a conventional black box that is notbased on the vector model.

Returning back to FIG. 8, an example will be now be explained withreference to the processing paths on the right hand side of FIG. 8,where vector model 404 is transmitted to an external location, forfurther processing and evaluation. In step 808, vector model 404generated at step 712 (FIG. 7) by the first processing unit 308 istransmitted by an antenna 304 (FIG. 3) to an external site, locatedremotely from the vehicle 300 (FIG. 3).

Transmission may be performed over a standard transmission channel, suchas radio or a cellular telephone line. Transmitting the vector model 404to the external location is not, however, limited to these. Anytransmission technique available at present or in the future can beemployed. For example, a sequence of vector models 404 representing athree-dimensional space-and-time model is transferred.

In subsequent step 810, the transmitted vector model 404 undergoesexternal processing. External processing may be performed by a computersystem provided at the external location. Various kinds of externalprocessing can be performed, including those which are generally similarto the processing evaluation performed internally in step 802 describedabove by second processing unit 310 (FIGS. 3 & 4). In one example of animplementation, external processing performed at step 810 also includesrendering the information from vector model 404 for display at anexternal computer. A remote location, where external processing isperformed may be a stationary location. Processing at a mobile location,such as a specialized vehicle, may also, or alternatively, be performed.Since computer systems available at a stationary location are generallymore powerful than the specialized processing units available on-boardthe vehicle, more sophisticated processing procedures can be generallyperformed at the remote location. Thereby, a response can be generatedin quasi-real time. The real time requirements can be particularlyfulfilled in view of the possibility of transmitting the processing databy a channel having a limited bandwidth, in the form of the vectormodels 404 having specifically reduced data volume.

At step 812, a processing result from external evaluation isre-transmitted to the vehicle. The same medium for transmission may beutilized for re-transmission, as for transmission of vector model 404from vehicle 300 to the remote location.

On the basis of the processing result re-transmitted in step 812, remotecontrol of the vehicle can be achieved (step 814). The processing resultis transmitted at step 812 to vehicle 300 in form of data representingprocessing results that have to be interpreted in step 814 at thevehicle for vehicle control. A person skilled in the art is aware thatalternatively a signal for vehicle control can be directly generated atthe remote location, and be transmitted to the vehicle so as to bedirectly used by a specifically adapted control unit located at thevehicle.

It is further noted that transmission of vector models 404 from vehicle300 to a remote location is not limited to the example explained above.Transmission of vector model information can be utilized for variousother purposes and objects, including visualization at an externalcomputer, for instance, to watch drivers undergoing training in a closedtraining area from a central site by a single instructor. Another areaof application of a vector model 404 transmitted to a remote location isimmediate notification of accidents or other extraordinary eventsoccurring with specialized vehicles, such as VIP cars, to a centralinstance.

FIG. 12 illustrates the overall structure of an object model 1202 thatmay be s employed for generating vector model 404 (FIG. 4) in accordancewith one example of an implementation of the invention. Object model1202 classifies a plurality of objects in the form of a hierarchicallyarranged tree-like structure into classes and sub-classes. Each classand sub-class corresponds to a particular level of the hierarchical treestructure. In FIG. 12, three levels of an object model are shown forillustration. In the example of object model 1202 illustrated in FIG.12, the number of illustrated levels are only given by way of example,and are not intended for limitation. Every object class/sub-class on aparticular level is associated with object classification information.The object classification information indicates that a particular objecthas to be distinguished from objects belonging to other classes at thesame hierarchical level. Examples of object information for three objectclass levels 1204, 1206, 1208 are given in the boxes in FIG. 1200.

FIG. 12 illustrates an example of an object model, having hierarchicalobject classification that is particularly adapted to objects occurringin a vehicle's environment, that are most likely to be captured by athree-dimensional camera, such as camera 302 a (FIG. 3), mounted on avehicle. If an object that is present in a captured image has beenrecognized as belonging to an entity of the object classes on aparticular level, a vector describing the respective object in thevector model is associated with the corresponding object classificationinformation. The object classification information serves as a pointerto data representing characteristic features of all objects of therespective class, that are available at the database 412 (FIG. 4), andcan be accessed when processing the respective object indicated by thevector model 404.

The information contained in the vector model 404 can thus be renderedmore detailed by increasing the number of levels of the object model. Inaccordance with the increasing number of sub-classes with a largeramount of levels, more and more detailed information can be provided tobe accessed from the vector model 404, in compliance with theclassification information of an object described by single or aplurality of vectors. On the other hand, the amount of information(e.g., reflected by a corresponding data volume) increases with anincreasing number of hierarchical levels as well. Therefore, on thebasis of the hierarchical level of the model, a trade-off betweeninformation content and data volume can be easily achieved by extendingor restricting the number of hierarchy levels, depending on the currentsituation.

In the case of the object model of FIG. 12, at the first hierarchy level1204, all objects are classified into vehicles and other objects. Onsecond hierarchy level 1206 of the illustrated object model, each classof level 1 is sub-divided into two sub-classes. Namely, the class ofvehicles comprises sub-classes of two-wheeled vehicles, and of vehicleshaving four and more wheels. The class of all other objects issub-divided into a sub-class of pedestrians, and a sub-class of allother objects.

A more detailed description of objects is enabled, if a vector modeladditionally includes information relating to object classification onthe third hierarchical level 1208. It is not, however, necessary for aparticular class or sub-class of a specific level to be furthersub-divided on the next level. For instance, the sub-class ofpedestrians of second level 1206 is not further sub-divided on thirdlevel 1208. The other exemplary sub-classes of second level 1206 arefurther sub-divided, where the number of sub-classes of third level 1208varies. Namely, the exemplary sub-class of two-wheeled vehicles issub-divided twice into the third level sub-classes 1208 of motorbikesand bicycles. The second level sub-class 1206 of vehicles with four andmore wheels is sub-divided into the three third level sub-classes 1208of cars, lorries and buses. The second level sub-class 1206 of otherobjects is sub-divided into five third level sub-classes 1208, ofcurbstones, trees, buildings, traffic signs and other objects. Asindicated on the right hand side of FIG. 12, further sub levels arepossible for some, or all, of the object sub-classes of third level1208. For instance, traffic signs can be classified into fixed trafficsigns, and traffic signs with a variable meaning (such as trafficlights).

In summary, a vehicle environment monitoring system is provided that isbased on a three-dimensional vector model 404 (FIG. 4). Thethree-dimensional vector model 404 of the vehicle's environment isgenerated on the basis of the image data captured by at least athree-dimensional camera. Out of the image data, particular data areextracted for generating the three-dimensional vector model 404 toreduce the data volume. For data extraction, a data extraction algorithm(step 704, FIG. 7) is applied that is determined in accordance with atleast one parameter that relates to the situation of the vehicle.Accordingly, the system specifically targets data extraction that isperformed for generating a three-dimensional model 404 particularlyadapted for an application that is desired in the current vehiclesituation. The applications of the vector model include driverassistance, external monitoring and vehicle control, as well asrecording in an event data recorder 406, to which a sequence ofthree-dimensional vector models, representing a three-dimensionalspace-and-time model, is generated.

In particular, the invention employs a situation-dependent extraction ofparticular data from a captured three-dimensional image of anenvironment of a vehicle. The amount and character of the data that areextracted are determined and adapted to the specific situation and theintended target of the environment monitoring. The particular dataextracted out of the image information are employed for generating athree-dimensional vector model of the vehicle environment from acaptured three-dimensional image. Thereby, a situation-dependentreduction of the complete information captured by a three-dimensionalcamera is enabled, and a high density of monitoring information may beachieved in a target-oriented manner.

In one example, the information content of the three-dimensional vectormodel may be made available for further processing. In particular, adriver assistance system can employ the three-dimensional vector modelfor generating measures for assisting a driver in response toenvironment information reflected in the 3D-vector model. Accordingly,the step of generating the 3D-vector model decouples the environmentdata collection from the intelligence for issuing a response in asituation adapted manner. Therefore, a trade-off between response timerequirements and accuracy of information content may be achieved.

For example, the data extraction algorithm may be adapted to theprocessing time available for generating the 3D-vector model.Accordingly, the system enables a situation adapted balancing betweenprocessing time and information content requirements. For instance, fordriver assistance of a vehicle driving at a high speed on a motorway, itmay be critical to have a response time as short as possible. As thesystem response time basically depends on the processing time, it istherefore advisable to extract only the most security relevant data, andto discard that data that is of no or only minor relevance for vehiclesecurity in generating the model, in order to save processing time. Onthe other hand, in a situation such as employing the model for assistingin parking a car, the response time is not of such critical importance.Therefore, the model generated for parking assistance can process moreinformation and thus achieve a higher accuracy, which may be helpfulwhen parking a car.

The system may further determine a maximum amount of processing timecorrelating to a specific vehicle situation. In which case, a maximumamount of data to be extracted can be further determined. The processingtime and the respective response time basically depend on the dataamount to be processed. It is thereby guaranteed that the response timecan be kept within an upper limit that is predetermined for the specificsituation.

To assist with processing time, the data extraction algorithm mayinclude object recognition. Object recognition enables therepresentation of features of predefined objects in an especially datasaving manner. The data extraction algorithm may include filtering thecaptured three-dimensional image. Accordingly, the image may bepre-processed to reduce the data amount and to avoid an undesiredinaccuracy.

In a further example, the data extraction algorithm may extract dataonly from a part of the three-dimensional image corresponding to acertain visual field. For instance, in a case of driving at a highspeed, the most relevant information may be contained in a visual fieldrepresenting a narrow cone-shaped sector in forward and backwarddirection of the driven car. In this case, it may be necessary toquickly grasp changes in the traffic situation, even those occurring ata far distance. On the other hand, for parking a car under conditions oflimited space, it may be helpful to have an all-round view, which can,however, be limited to a narrow range of distance.

The extraction algorithm may further be determined taking into account apriority for different kinds of data to be extracted from thethree-dimensional image. For different vehicle situations and fordifferent purposes of employing the three-dimensional vector model,different kinds of data are of importance, while other data are lessimportant or can be completely disregarded. Under the condition of alimited amount of data to be processed, it is therefore desirable notonly to define an upper limit of the data amount, but also to definepriorities of the different kinds of data to be extracted, until thelimit of data amount is reached. For instance, for collision warning ona motorway, most relevant kinds of data are positions and relativevelocity to other vehicles on the motorway. On the other hand, for aparking assistance, detailed object information of all objects isrequired, but only in a low distance range around the vehicle.

Accordingly, the vehicle situation may be analyzed based on the capturedthree-dimensional image itself, in order to obtain a parametercharacterizing the current vehicle situation. Object recognition can,for instance, determine based on an image, whether a vehicle is beingdriven in dense traffic or not. Also environmental parameters such asthe width of the road can be easily recognized from an image.

Detecting vehicle status information in addition to the informationcaptured by the three-dimensional camera provides a value of a parameterfor characterizing the vehicle situation. In particular, the detectedvehicle status information may include velocity information of thevehicle. Accordingly, the parameters that are generally measured in avehicle can be employed for determining the data extraction algorithm,without any further effort. The size of the visual field from which thedata extraction algorithm extracts data is the smaller the higher thevelocity of the vehicle. Accordingly, information from a larger distancecan be taken into account while not enlarging the overall data amount.

The environment monitoring system may be further capable of taking intoaccount a parameter value that has been received from an inputinterface. For instance, a switchover may be enabled between a parkingassistant mode and a mode for assisting a driver in dense traffic. Thedetermination of the extraction algorithm then includes definingvariables of an algorithm. For instance, filtering parameters forfiltering a complete image of a part of an image can be adjusted inaccordance with accuracy and data amount requirements. Further, aparticular one of a plurality of available data extraction algorithmsmay be selected on the basis of a parameter describing the vehiclesituation.

As discussed above, the three-dimensional vector model representsobjects in the environment of the vehicle by vectors and objectspecification information. The object specification informationassociates an object with standard object classes of an object model.Employing an object model may assist in connection with objectrecognition. A recognized object is associated with standard objectinformation of a database. It is therefore not necessary to processinformation once more every time when a new object belonging to one ofthe standard classes occurs.

When utilizing an object model, the object model is stored in adatabase. A database enables the pre-storing of standard information ofsuch objects that are most likely to occur and most relevant to thevehicle's environment, such as other vehicles, persons and trafficsigns.

In another example, the system may generate a sequence ofthree-dimensional vector models during a certain period of time.Thereby, a three-dimensional space-and-time model is obtained. Moreover,a co-ordinate system fixed to the vehicle may be used as a referencesystem for the vectors of the sequence of the three-dimensional vectormodels. Employing a vehicle centered reference system may enable themost natural view of all modifications occurring in the vehicle'senvironment with time, as only the changes occurring relative to thevehicle are of importance.

In yet another example of a system, a response to the information of thevehicle environment represented by the three-dimensional space-and-timemodel is generated. Accordingly, the three-dimensional space-and-timemodel is employed to realize functions for assisting a driver, ordirectly interfering with vehicle control. For example, a generatedresponse is directed towards avoiding an emergency situation such as thethreat of a collision or a loss of the driver's control over hisvehicle. The response may comprise a notification of an indication of anemergency situation to be expected to the driver. The notification canbe issued in the form of an acoustic signal and/or a warning on adisplay within the field of vision of the driver. Further, a device forassisting a driver to avoid an emergency situation may be notified. Aresponse generated on the basis of the three-dimensional space-and timemodel may include an automatic interference with vehicle control. Forinstance, the system of the invention may enable automatic braking, ifany notifications of collision warning have been ignored for apredetermined time, or if it is judged, on the basis of thethree-dimensional space-and time model that the probability of acollision has become larger than a predefined threshold.

The three-dimensional vector model may be further processed forassisting a driver in controlling the vehicle based on the processingresult. For example, the generated response may be directed towardsassisting a driver in parking the vehicle. Such responses may includetaking countermeasures to minimize the consequences of a threateningaccident. In particular, the counter measures may include status controlof vehicle safety equipment, such as an airbag or a seatbelt. Forinstance, if the three-dimensional space-and time model indicates that acollision is likely to occur, an airbag can be controlled to betransferred in a pre-triggered status. Accordingly, the airbag can bereleased faster, if the collision indeed occurs. On the other hand, ifit is nevertheless possible to still avoid the collision, the airbag canbe returned to the normal status, and remains unused.

While the illustrated example shows a three-dimensional image capturedfrom at least one camera. The system may capture a plurality ofthree-dimensional images from plural camera positions. More preferably,the plurality of three-dimensional images may represent an environmentarea including a predetermined range around the vehicle. As the field ofvision of a single camera is generally limited, a plurality of camerascan provide more complete information about the environment of thevehicle. Alternatively, a position of a single camera can be controlledso as to provide a different view. For instance, a camera can be pannedaround.

In addition to image data to be captured by a camera, also data fromfurther sensors a vehicle is equipped with are taken into account forgenerating the three-dimensional vector model. For example, forgenerating the three-dimensional vector model, a process of sensorfusion may be employed to transform sensor data into a unique format.During sensor fusion, moreover, data acquired from different sensors ischecked for inconsistencies. Inconsistent and redundant data is removedduring sensor fusion. Image data is included into the process of sensorfusion. Such further sensors employed for generating thethree-dimensional vector model may include a motion sensor, anaccelerometer, a temperature sensor, a radar sensor and/or an infraredsensor.

In one example implementation of a system, as explained in detail above,the three-dimensional vector model is transmitted to an externallocation with respect to the vehicle, which external location may beremote or stationary. A sequence of three-dimensional vector modelsrepresenting a three-dimensional space-and-time model is transmitted tothe external location. For example, the three-dimensional vector modelmay be transmitted by radio, by cellular telephone line, or otheravailable communication transmission methods. The three-dimensionalvector model may be transmitted for visualizing on an external computer.Accordingly, for instance, an instructor of a driving school can watch aplurality of vehicles undergoing training in a training area.

When transmitted externally, the three-dimensional vector model may befurther evaluated in an external computer. The results of the evaluationmay then be transmitted back to the vehicle. More preferably, theevaluation results are employed for remotely controlling the vehicle.Either, the re-transmitted evaluation results are utilized forgenerating and issuing the control signal at the vehicle itself.Alternatively, a control signal can be generated at the externallocation, so that only the control signal itself is re-transmitted tothe vehicle.

As the bandwidth available for transmission of signals between a drivenvehicle and an external location is generally limited, the system of theinvention enables maximum employment of the available bandwidth via amaximally targeted selection of data to be extracted for generating thethree-dimensional vector model.

In another example implementation, the three-dimensional vector model issaved in an event data recorder. An event data recorder enablesreconstruction of crash details in the case of an accident. A sequenceof three-dimensional vector models that has been generated during apredetermined period of time is saved into the event data recorder. Thethree-dimensional vector models of the sequence may be associated withtime stamps before saving. Still more preferably, the three-dimensionalvector models are saved for a predetermined storing time, whichapproximately corresponds to an available storage capacity. Only in thecase that an impact is detected that is most probably caused by anextraordinary event such as an accident, the three-dimensional vectormodels remain permanently saved, while further storage of newlygenerated vector models is inhibited. The three-dimensional vectormodels may also be stored in a cyclic storage having a predeterminedstorage capacity, and are cyclically overwritten, as long as no impactis detected.

Targeted extraction of data from complete information enabled by camerasattached to a vehicle generally reduces the amount of data to be storedfor a certain instance of time. Accordingly, for a storage capacity ofan event data recorder, a history that can be stored becomes much longerwhen compared with a conventional camera based event data recorderstoring complete image information.

It will be understood, and is appreciated by persons skilled in the art,that one or more processes, sub-processes, or process steps described inconnection with FIGS. 4-11 may be performed by a combination of hardwareand software. The software may reside in software memory internal orexternal to the signal processor 308, 310 or other controller, in asuitable electronic processing component or system such as, one or moreof the functional components or modules schematically depicted in FIGS.3, 4 and 5. The software in software memory may include an orderedlisting of executable instructions for implementing logical functions(that is, “logic” that may be implemented either in digital form such asdigital circuitry or source code or in analog form such as analogcircuitry or an analog source such an analog electrical, sound or videosignal), and may selectively be embodied in any tangiblecomputer-readable medium for use by or in connection with an instructionexecution system, apparatus, or device, such as a computer-based system,processor-containing system, or other system that may selectively fetchthe instructions from the instruction execution system, apparatus, ordevice and execute the instructions. In the context of this disclosure,a “computer-readable medium” is any means that may contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer readable medium may selectively be, for example, but is notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, device, or medium. More specificexamples, but nonetheless a non-exhaustive list, of computer-readablemedia would include the following: a portable computer diskette(magnetic), a RAM (electronic), a read-only memory “ROM” (electronic),an erasable programmable read-only memory (EPROM or Flash memory)(electronic), and a portable compact disc read-only memory “CDROM”(optical) or similar discs (e.g. DVDs and Rewritable CDs). Note that thecomputer-readable medium may even be paper or another suitable mediumupon which the program is printed, as the program can be electronicallycaptured, via for instance optical scanning of the paper or othermedium, then compiled, interpreted or otherwise processed in a suitablemanner if necessary, and then stored in a computer memory.

Although the previous description only illustrates particular examplesof various implementations, the invention is not limited to theforegoing illustrative examples. A person skilled in the art is awarethat the invention as defined by the appended claims can be applied invarious further implementations and modifications. In particular, acombination of the various features of the described implementations ispossible, as far as these features are not in contradiction with eachother. Accordingly, the foregoing description of implementations hasbeen presented for purposes of illustration and description. It is notexhaustive and does not limit the claimed inventions to the precise formdisclosed. Modifications and variations are possible in light of theabove description or may be acquired from practicing the invention. Theclaims and their equivalents define the scope of the invention.

1. A method of monitoring the environment of a camera-equipped vehicle,comprising the steps of: capturing a 3D-image of the environment of thevehicle, the 3D-image representing a predetermined area of the vehicleenvironment; reducing the amount of information acquired in said imagecapturing step by extracting data from said 3D-image employing a dataextraction algorithm; and generating a 3D-vector model of the vehicleenvironment from said data extracted from said 3D-image characterized bythe step of: determining the data extraction algorithm to be employed insaid reducing step based on at least one parameter characterizing thevehicle situation.
 2. The method of claim 1, where said extractionalgorithm determining step determines the data extraction algorithm tobe adapted according to the processing time available for generating the3D-vector model.
 3. The method of claim 1, where said extractionalgorithm determining step determines the data extraction algorithm tobe adapted according to information reflected in the 3D-vector model. 4.The method of claim 1, where said data extraction algorithm includesobject recognition.
 5. (canceled)
 6. The method of claim 1, where saiddata extraction algorithm extracts data only from a part of the 3D-imagecorresponding to a certain visual field.
 7. The method of claim 1,further comprising the step of further processing said 3D-vector modelfor assisting a driver in controlling the vehicle based on theprocessing result.
 8. The method of claim 7, where said extractionalgorithm determining step includes the step of determining a maximumavailable amount of time for performing said steps of generating andfurther processing said 3D-vector model based on the vehicle situationand further takes into account said maximum available amount of time todetermine said data extraction algorithm.
 9. (canceled)
 10. The methodof claim 9, where said extraction algorithm determining step includesthe step of determining a maximum amount of data to be extracted incorrespondence with said maximum amount of time.
 11. The method of claim10, where said extraction algorithm determining step further includesthe step of determining priorities for different kinds of data that canbe extracted from said 3D-image based on the particular relevance ofdifferent kinds of data in the vehicle situation characterized by saidat least one parameter.
 12. The method of claim 1, where said extractionalgorithm determining step further includes the step of analyzing thevehicle situation based on the captured 3D-image, where a value of saidat least one parameter is obtained from said analyzing step.
 13. Themethod of claim 1, further comprising the step of detecting vehiclestatus information, where the detected vehicle status informationincludes a value of said at least one parameter.
 14. The method of claim13, where said vehicle status information includes velocity informationof said vehicle, and where the size of the visual field from which saiddata extraction algorithm extracts data decreases as the velocity of thevehicle increases.
 15. (canceled)
 16. (canceled)
 17. The method of claim1, where said extraction algorithm determining step includes the step ofselecting a particular data extraction algorithm out of a plurality ofdata extraction algorithms based on a parameter describing the vehiclesituation.
 18. The method of claim 1, where said 3D-vector modelrepresents objects in the environment of the vehicle by vectors andobject classification information, said object classificationinformation associates an object with standard object classes of anobject model.
 19. The method of claim 18, where said object model ispre-stored in a database containing standard information of such objectsthat are most likely to occur and most relevant to the vehicle'senvironment.
 20. (canceled)
 21. The method of claim 1, furthercomprising the step of: generating a 3D-space-and-time model, where said3D-space-and-time model includes a sequence of 3D-vector models. 22.(canceled)
 23. The method of claim 21, further comprising the step ofgenerating a response to the information of the vehicle environmentrepresented by said 3D-space-and-time model.
 24. The method of claim 23,where said response beg is directed towards avoiding an emergencysituation to be expected by a driver.
 25. (canceled)
 26. (canceled) 27.The method of claim 24, where said response includes an automaticinterference with the vehicle control.
 28. The method of claim 23, wheresaid response is directed towards assisting a driver in parking thevehicle.
 29. The method of claim 23, where said response includes takingcountermeasures to minimize the consequences of a threatening accident.30. (canceled)
 31. The method of claim 1, where said image capturingstep captures a plurality of 3D-images from various camera positions,where said plurality of 3D-images re-present an environment areaincluding a predetermined range around the vehice.
 32. (canceled) 33.The method of claim 1, where said 3D-vector model generating stepfurther employs data from sensors.
 34. The method of claim 33, furthercomprising the step of sensor fusion to transform sensor data into aunique format and remove inconsistent and/or redundant data. 35.(canceled)
 36. The method of claim 1, further comprising the step oftransmitting the 3D vector model to an external location with respect tosaid vehicle.
 37. (canceled)
 38. The method of claim 36, where said3D-vector model is transmitted via a wireless communication.
 39. Themethod of claim 36, where said 3D-vector model is transmitted to andvisualized on an external computer.
 40. The method of claim 36, furthercomprising the step of evaluating said 3D-vector model at an externalcomputer.
 41. The method of claim 40, further comprising the step ofre-transmitting results of said evaluation back to the vehicle.
 42. Themethod of claim 40, further comprising the step of remotely controllingthe vehicle by employing the evaluation results.
 43. (canceled)
 44. Themethod of claim 36, where said transmitting step transmits a sequence of3D-vector models representing a 3D-space-and-time model.
 45. The methodof claim 1, further comprising the step of saving the 3D-vector model inan event data recorder.
 46. The method of claim 45, further comprisingthe steps of: generating a sequence of 3D-vector models during apredetermined period of time; and saving said sequence of 3D-vectormodels into the event data recorder.
 47. The method of claim 46, furthercomprising the step of associating the 3D-vector models with time stampsbefore saving into the event data recorder.
 48. The method of claim 47,where the 3D-vector models are saved for a predetermined storing time;the method further comprising the step of detecting an impact caused byan extraordinary event, where the 3D-vector models remain permanentlysaved upon said detection of an impact.
 49. The method of claim 48,where said 3D-vector models are stored in a cyclic storage having apredetermined storage capacity; the method further comprising the stepof cyclically overwriting stored 3D-vector models having the earliesttime stamp, until said detection of au impact.
 50. An environmentmonitoring device for monitoring the environment of a vehicle,comprising: a 3D-camera for capturing an image of the environment of thevehicle, the image representing a predetermined area of the vehicleenvironment; and a first processing unit for processing the informationacquired by said 3D-camera, said first processing unit including: a dataextracting unit for extracting data from said image by employing a dataextraction algorithm; and a model generation unit for generating a3D-vector model of the vehicle environment from said data extracted fromsaid image; characterized in that said first processing unit furtherincludes a determining unit for determining the data extractionalgorithm to be employed by said data extracting unit based on at leastone parameter characterizing the vehicle situation.
 51. The environmentmonitoring device of claim 50, where said determining unit determinesthe data extraction algorithm to be adapted according to the processingtime available for generating the 3D-vector model.
 52. The environmentmonitoring device of claim 50, where said determining unit determinesthe data extraction algorithm to be adapted according to informationreflected in the 3D-vector model.
 53. The environment monitoring deviceof claim 50, where said data extraction algorithm includes objectrecognition.
 54. (canceled)
 55. The environment monitoring device ofclaim 50, where said data extraction algorithm extract data only from apart of the image corresponding to a certain visual field.
 56. Theenvironment monitoring device of claim 50, further comprising a secondprocessing unit for further processing said 3D-vector model forassisting a driver in controlling the vehicle based on the processingresult.
 57. The environment monitoring device of claim 56, where saiddetermining units determines a maximum available amount of time forgenerating and further processing said 3D-vector model based on thevehicle situation, and further takes into account said maximum availableamount of time for determining said data extraction algorithm. 58.(canceled)
 59. The environment monitoring device of claim 58, where saiddetermining unit determines a maximum amount of data to be extracted incorrespondence with said maximum amount of time.
 60. The environmentmonitoring device of claim 59, where said determining unit furtherdetermines priorities for different kinds of data that can be extractedfrom said image based on the particular relevance of different kinds ofdata in the vehicle situation characterized by said at least oneparameter.
 61. The environment monitoring device of claim 50, where saiddetermining unit further analyzes the vehicle situation based on thecaptured image for obtaining a value of said at least one parameter. 62.The environment monitoring device of claim 50, further comprising adetector for detecting vehicle status information, where the detectedvehicle status information includes a value of said at least oneparameter.
 63. The environment monitoring device of claim 62, where saidvehicle status information includes velocity information of saidvehicle, and where the size of the visual field from which said dataextraction algorithm extracts data decreases as the velocity of thevehicle is increases.
 64. (canceled)
 65. (canceled)
 66. The environmentmonitoring device of claim 50, where said determining unit selects aparticular data extraction algorithm out of a plurality of dataextraction algorithms based on a parameter describing the vehiclesituation.
 67. The environment monitoring device of claim 50, where said3D-vector model represents objects in the environment of the vehicle byvectors and object classification information, said objectclassification information associates an object with standard objectclasses of an object model.
 68. The environment monitoring device ofclaim 67, further comprising a database for pre-storing said objectmodel.
 69. The environment monitoring device of claim 67, where saidextraction algorithm obtains object classification information from theimage.
 70. The environment monitoring device of claim 50, where saidmodel generation unit generates a 3D-space-and-time model that includesa sequence of 3D-vector models.
 71. (canceled)
 72. The environmentmonitoring device of claim 70, further comprising a second processingunit for generating a response to the information of the vehicleenvironment represented by said 3D-space-and-time model.
 73. Theenvironment monitoring device of claim 72, where said response isdirected towards avoiding an emergency situation to be expected by adriver.
 74. The environment monitoring device of claim 73, furthercomprising a device for notifying a driver of an expected emergencysituation based on the response generated by said second processingunit.
 75. The environment monitoring device of claim 73, furthercomprising a device for assisting a driver to avoid an emergencysituation based on the response generated by said second processingunit.
 76. The environment monitoring device of claim 75, where saiddevice is a controller for automatically interfering with the vehiclecontrol based on the response generated by said second processing unit.77. The environment monitoring device of claim 72, where said responseis directed towards assisting a driver in parking the vehicle.
 78. Theenvironment monitoring device of claim 72, further comprising acontroller for taking countermeasures to minimize the consequences of athreatening accident based on the response generated by said secondprocessing unit.
 79. (canceled)
 80. The environment monitoring device ofclaim 50, comprising a plurality of 3D cameras for capturing images fromvarious camera positions, where the images captured by said plurality of3D-cameras representing an environment area include a predeterminedrange around the vehicle.
 81. (canceled)
 82. The environment monitoringdevice of claim 50, further comprising sensors, where said modelgeneration unit further employs data from said sensors for generatingsaid 3D-vector model.
 83. The environment monitoring device of claim 82,further comprising a sensor fusion unit for transforming sensor datainto a unique format and removing inconsistent and/or redundant data.84. (canceled)
 85. The environment monitoring device of claim 50,further comprising a transmitter for transmitting the 3D-vector model toan external location with respect to said vehicle.
 86. (canceled) 87.The environment monitoring device of claim 85, where said transmittertransmits said 3D-vector model via wireless communication.
 88. Theenvironment monitoring device of claim 85, where said transmittertransmits said 3D-vector model for visualizing on an external computer.89. The environment monitoring device of claim 85, where saidtransmitter transmits said 3D-vector model for evaluating at an externalcomputer.
 90. The environment monitoring device of claim 89, furthercomprising receiving unit for receiving results of said evaluationtransmitted back to the vehicle.
 91. The environment monitoring deviceof claim 89, further comprising a controller for controlling the vehicleby employing the evaluation results received from the external computer.92. (canceled)
 93. The environment monitoring device of claim 85, wheresaid transmitter transmits a sequence of 3D-vector models representing a3D-space-and-time model.
 94. The environment monitoring device of claim50, further comprising an event data recorder for saving the 3D-vectormodel.
 95. The environment monitoring device of claim 94, where saidmodel generation unit generates a sequence of 3D-vector models during apredetermined period of time, and where said sequence of 3D-vectormodels is saved into the event data recorder.
 96. The environmentmonitoring device of claim 95, where the 3D-vector models are associatedwith time stamps before saving.
 97. The environment monitoring device ofclaim 96, further comprising an impact detector for detecting an impactcaused by an extraordinary event, where the 3D-vector models are savedin the event data recorder for a predetermined storing time and the3D-vector models remain permanently saved when an impact is detected bysaid impact detector.
 98. (canceled)